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(54) ROBOT MONITORING DEVICE 
(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a monitoring device for monitoring the 
operation of a robot by a system different from robot control concerned. 
SOLUTION: The watching ranges 14 for watching the invasion of a robot are preset 
near robots 12, the watching ranges 14 are picked up for each specified time by using 
image pickup means 16, and whether a moving object exists in a part corresponding to 
the watching range 14 in an image picked up by the image pickup means 16 is judged 
by an analyzing means. The analyzing means judges the presence or absence of a 
moving object by detecting the movement of a part corresponding to the watching 
range 14 in the image. When it is judged that a moving object exists in the watching 
range 14, supply of power source to the robot 12 is stopped, or warning sound and/or 
warning light is emitted by providing a warning means for emitting warning sound 
and/or warning light on a monitoring device, and therefore, generation of abnormalities 
is preliminarily prevented. 



* NOTICES * 



JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect 
the original precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1]A robot monitoring instrument comprising: 

An imaging means (16) which photos a watch area (14) which looks out for invasion of 
a robot set up near the robot (12) for every predetermined time. 
An analysis means (18) to distinguish whether a movable matter object exists in a 
portion equivalent to a watch area in a picture photoed by this imaging means. 

[Claim 2]The robot monitoring instrument according to claim 1 which an analysis 
means (18) detects difference lost motion of image data of an image region equivalent 
to a watch area (14) during the minute time delta t, and distinguishes existence of a 
movable matter object in a watch area. 

[Claim 3]The robot monitoring instrument according to claim 2 which detects a motion 
of a movable matter object and distinguishes existence of a movable matter object 
when an analysis means (18) generates a difference lost-motion vector of image data 
of an image region equivalent to a watch area (14) during the minute time delta t. 
[Claim 4]The robot monitoring instrument according to claim 1 to 3 which transmits a 
command made to suspend supply of a power supply to a robot (12) when a movable 
matter object existed all over a watch area (14) and an analysis means (18) is 
distinguished. 

[Claim 5]When an analysis means (1 8) was connected to an alarming means (44) which 
emits a warning sound and/or alarm light, a movable matter object existed all over a 
watch area (14) and an analysis means (18) distinguishes, The robot monitoring 
instrument according to claim 1 to 4 which transmits a command which emits a 
warning sound and/or alarm light to said alarming means (44). 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the device which supervises robots, 

such as an industrial robot. 

[0002] 

[Description of the Prior Art]Generally robots, such as an industrial robot, equip an 
inside with monitoring programs, such as a watchdog timer which controls operation, 
and the fault of an operation program, disturbance, etc. by a cause. When a robot 
separates from desired operation and it operates, it judged with a "reckless run", and 
operation of a robot is stopped or the alarm is emitted. 
[0003] 

[Problem(s) to be Solved by the Invention] However, when the monitoring program and 
the program which controls operation of a robot were controlled by the same CPU in 
many cases and an error arose in CPU, even if the robot hung up, the monitoring 
program might not fully operate. Since a monitoring program was what supervises 
operation of the robot itself, even if movable matter objects other than the robots 
concerned, such as a person and other robots, approached the robot, it has not 
recognized it and was not able to prevent the collision with the movable matter object 
which approaches, and the robot concerned. 

[0004]The purpose of this invention supervises operation of a robot by a different 
system, when a robot deviates from predetermined workspace and operates, stop 
supply of the power supply to a robot, stop operation of a robot, or, When other 
movable matter objects approach a robot, it is providing the monitoring instrument 
which can emit an alarm and can prevent the collision of a movable matter object and 
a robot. 
[0005] 

[Means for Solving the Problem]In order to solve an aforementioned problem, a robot 
monitoring instrument (10) of this invention, A watch area (14) which looks out for 
invasion of the robot concerned near the robot (12) is set up beforehand, It is 
distinguished by an analysis means (18) whether movable matter objects including the 
robot (12) concerned exist in a portion equivalent to a watch area (14) in a picture 
which photoed this watch area (14) for every predetermined time using an imaging 
means (16), and was photoed by an imaging means (16). An analysis means (18) 
distinguishes existence of a movable matter object by detecting a motion of a portion 
equivalent to a watch area (14) in a screen. For example, when a motion vector of a 
portion equivalent to a watch area (14) in a screen is generated and a size in a watch 
area distinguishes existence of a motion vector which is not zero, a movable matter 
object can be recognized. The following control is performed, when a movable matter 



object existed in a watch area (14) and an analysis means (18) distinguishes. For 
example, when the invasion direction of a movable matter object aims to go to a watch 
area from the robot side, a movable matter object which trespassed upon a watch 
area (14) is the robot (12) concerned, and since it is thought that a robot (12) has 
hung up, supply of a power supply to a robot (12) is stopped. When the invasion 
direction of a movable matter object aims to face to a robot, since movable matter 
objects are a person and other robots, they prevent generating of abnormalities by 
providing an alarming means (44) which emits a warning sound and/or alarm light in a 
monitoring instrument (10), and emitting a warning sound and/or alarm light. 
[0006] 

[Function and Effect]The robot monitoring instrument (10) of this invention monitors 
continuously the watch area (14) beforehand set up near the robot (12), detects a 
motion of the picture in a watch area (14), and generates a motion vector. It can ask 
for an objective motion and motion vector by calculating the difference of image data, 
such as a luminance signal in the continuous 2 inter-frame watch area in minute time 
deltat, for example. When a motion of an object is detected all over a watch area (14) 
(i.e., when the picture in which a size has a motion vector which is not zero is 
detected), it considers that a robot (12) and other movable matter objects have 
trespassed upon the watch area (14), and supply of the power supply to a robot is 
suspended, or an alarm etc. are emitted. 

[0007]According to this invention, even when a robot hangs up and fault arises in the 
monitoring program inside a robot, since the robot is supervising the robot by the 
separate system, it can prevent a reckless run etc. which became uncontrollable by 
the robot itself. It can supervise not only about the surveillance of only a robot but 
about invasion of objects, such as a person to the inside of a watch area, and other 
robots, and a collision etc. can be beforehand prevented by emitting an alarm etc. 
[0008] 

[Embodiment of the Invention]The robot monitoring instrument (10) of this invention 
is provided with the imaging means (16) which photos the watch area (14) set up near 
the robot (12), and an analysis means (18) to analyze the picture photoed by the 
imaging means (16) as shown in drawing 1 and drawing 2 . 

[0009]Imaging devices, such as a video camera and a digital still camera, can be used 
for an imaging means (16), and it is arranged so that the neighborhood of the industrial 
robot (12) arranged along lines, such as a factory which performs the assembly of a 
product on a band conveyor (19), can be supervised from the upper part. Drawing 3 A 
thru/or D expresses the picture photoed by the imaging means (16). The field which 
can operate a robot (12), respectively is set to the circumference of an industrial 
robot (12). 

The watch area (14) which looks out for invasion of a robot is set to the 
circumference of a feasible region of operation. 



In this example, the watch area (14) of a robot (12) is set up as a back field and its 
headroom from the side of a robot. 

[0010]An imaging means (16) photos the picture (refer to drawing 3 A thru/or D) 
which includes a watch area in every minute time deltat, and transmits a picture signal 
to an analysis means (18). The pixel number of an imaging means (16) is made into sxt 
(however, s and t both positive integer). As for minute time deltat, it is desirable to set 
it as the 1 field, the scan time of one frame, or its integral multiple, for example, it can 
be made into 1 / 60 seconds. 

[0011]The picture photoed by the imaging means (16) is inputted into an analysis 
means (18). The AD translation of the picture signal inputted into the analysis means 
(18) is carried out by the A/D converter (20). Next, the luminance signal of only the 
pixel corresponding to a watch area (14) is taken out from the luminance signal Y in 
the picture signal by which the AD translation was carried out. As a method of taking 
out only the luminance signal of a watch area portion, for example the luminance signal 
Y. It has a storage parts store of the sxt individual corresponding to each pixel of the 
picture, and the method which "0" integrates by the data and the AND gate (22) of 
the field distinction memory (24) written in beforehand can be mentioned to the 
storage parts store corresponding to the watch area (14) set up into the picture at 
"1" and the other storage parts store. By integrating with the stored data in which a 
field distinction memory (24) corresponds by an AND gate (22), luminance signals 
other than a watch area are outputted as zero. The luminance signal which passed the 
AND gate (22) is memorized by the luminance memory (25). The luminance memory 

(25) has a storage parts store of the sxt individual corresponding to each pixel of the 
picture. 

The luminance signal inputted is memorized in order. 

[0012]Next, from an imaging means (16), it is transmitted to an analysis means (18) 
and the AD translation of luminance-signal Y' of the picture after minute time deltat 
progress is carried out similarly. Inputted luminance-signal Y' passes an AND gate (22), 
and luminance signals other than a watch area serve as zero similarly, and it is 
outputted. 

[0013]Next, the difference of the luminance signal memorized by the luminance 
memory and the luminance signal after time deltat is calculated in a difference circuit 

(26) , and change of luminosity is detected in the picture in a watch area (14). The 
difference of the luminosity calculated in the difference circuit (26) is 
absolute-value-ized in an absolute value-ized circuit (27), and when a size is not zero 
and "1" and a size are zero, it is distinguished from "0" in a binarization circuit (28). 
The picture element data in the watch area by which binarization was carried out is 
transmitted to a motion discrimination circuit (30), and the existence of the existence 
of a movable matter object is distinguished. 



[0014]For example, as shown in B from drawing 3 A, when there is no change of a 
picture all over a watch area (18) (the motion may arise in portions other than a watch 
area as shown in a figure), That is, when a movable matter object does not exist in a 
watch area, all are zero, and the difference of the luminance signal in a watch area 
serves as zero altogether, as the result which was absolute-value-ized and was 
carried out as for binarization is also shown in drawing 3 A-1. This result is 
transmitted to a motion discrimination circuit (30). About all the pixels of a watch area, 
since the result of an operation is "0", a motion discrimination circuit (30) judges that 
a motion does not exist all over a watch area. When a motion discrimination circuit 
judges "with no movable matter object", difference calculation of luminosity is 
performed about the following minute time deltat. 

[0015]As shown in D from drawing 3 B, the case where the tip of the arm (31) of a 
robot (12) invades all over a watch area (14) is considered. Since change has arisen in 
the luminance signal of the applicable portion in a watch area (14) about drawing 3 B 
and C if the difference of a luminance signal is calculated in a difference circuit (26), 
the difference does not serve as zero. If this difference calculation result is 
absolute-valueHzed and binarization is carried out in an absolute value-ized circuit 
(27), as shown in drawing 3 B-1 , only the portion into which the arm invaded will be set 
to "1", and the other portion will be set to "0." Since a motion will exist all over a 
watch area by this, a motion discrimination circuit (30) is judged to be "those with a 
movable matter object." When a motion judging circuit (30) judges it as "those with a 
movable matter object", mask processing of the data value of the inside by which 
binarization was carried out is carried out in a mask circuit (32) in the portion (33) 
surroun ded by "1", and it is transmitted to a centroid calculation circuit (34). A 
centroid calculation circuit (34) calculates the center of gravity of mask parts, and 
memorizes the coordinates in a center-of-gravity memory (36). As shown in drawing 3 
C and D next, when the arm (31) of a robot (12) invades further all over a watch area 
(14), ask for the difference of a luminance signal similarly and by the same processing 
as the above. As shown in drawing 3 C~1, the data value by which binarization was 
carried out carries out mask processing of the portion (33) surrounded by "1 ", and the 
barycentric coordinates are calculated. The barycentric coordinates searched for and 
the barycentric coordinates memorized by the center-of-gravity memory (36) are 
transmitted to a center-of-gravity difference circuit (38), and the difference between 
barycentric coordinates is called for. The difference of the coordinates searched for 
by this difference calculation serves as a motion vector (50) of the center of gravity of 
a movable matter object, and ( drawing 3 D~1 reference). The called-for motion vector 
is transmitted to a direction discrimination circuit (40). a direction discrimination 
circuit (40) takes out only the direction component in a motion vector, and the move 
direction of a movable matter object aims (following "for attitude") to separate from a 
robot (12) — or it is distinguished whether it aim (following "ON direction") to face to 



a robot (12). 

[0016]When a direction discrimination circuit (40) judges [ aiming (for attitude) to 
separate from a robot (12), and ] the move direction of a movable matter object 
Usually, as shown in D from drawing 3 B, since [ by which a movable matter object 
trespasses upon a watch area (14) from the robot side ] it is thought that it moves, 
that is, is a reckless run of a robot (12), a control circuit (42) intercepts the power 
supply circuit (43) of a robot (12), and supply of the power supply to a robot (12) is 
stopped. On the other hand, when a direction discrimination circuit (40) judges 
[ aiming (the ON direction) to face to a robot (12), and ] the move direction of a 
movable matter object, movable matter objects, such as a person or other robots, are 
the motions which trespass upon a watch area (14). 

Since it is not a reckless run of a robot (12), a control circuit (42) transmits a signal to 
an alarming means (44). 

An alarming means (44) emits an alarm with the command from a control circuit (42) 
on the movable matter object which invades by light or a sound. Thereby, the collision 
with a robot and a movable matter object can be prevented. 

[0017]When it is judged as "those with a movable matter object" irrespective of the 
invasion direction of a movable matter object in a motion discrimination circuit, the 
current supply to a robot may be suspended or it may be made to emit an alarm in the 
above-mentioned example, although the invasion to the watch area of a movable 
matter object from the motion vector in a watch area and its invasion direction were 
detected. 

[001 8]It may ask for a motion vector using a representative point matching method, 
the gradient method (gradient method), etc. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is a circuit diagram of the robot monitoring instrument of this invention. 
[Drawing 2] It is an explanatory view showing the state where the robot is supervised 
using an imaging means. 

[Drawing 3] It is an explanatory view showing the difference of the picture of an 

imaging means, and the luminosity in a watch area. 

[Description of Notations] 

(10) Monitoring instrument 

(12) Robot 

(14) Watch area 

(16) Imaging means 



(18) Analysis means 
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